BigQuery: Parse WITH CONNECTION on CREATE EXTERNAL TABLE by 97nitt · Pull Request #2326 · apache/datafusion-sqlparser-rs

97nitt · 2026-04-28T15:01:54Z

Summary

Parse the WITH CONNECTION <name> clause on BigQuery CREATE EXTERNAL TABLE
statements. Per the BigQuery DDL reference, external tables (and
external Iceberg tables in particular) carry a connection identifier between
the column list and the OPTIONS(...) clause:

CREATE OR REPLACE EXTERNAL TABLE `proj.ds.tbl`
WITH CONNECTION `projects/proj/locations/us/connections/c`
OPTIONS(format = "ICEBERG", uris = ["gs://b/m.json"]);

Today the parser rejects this shape because nothing consumes WITH CONNECTION,
and OPTIONS(...) on external tables only worked via the Hive
TBLPROPERTIES fallback path.

Changes

CreateTable / CreateTableBuilder: new with_connection: Option<ObjectName>
field, populated by the parser and rendered by Display between
CLUSTER BY and OPTIONS(...).
parse_create_external_table: consume an optional WITH CONNECTION <name>,
then prefer OPTIONS(...) over TBLPROPERTIES(...) (mutually exclusive in
practice; preserves the existing Hive path when neither is present).
Tests: new BigQuery cases covering WITH CONNECTION with OPTIONS,
WITH CONNECTION alone (Display normalizes by adding an empty column list),
the (columns) WITH CONNECTION OPTIONS(...) round-trip, and a baseline
that asserts with_connection stays None when the keyword pair is absent.
tests/sqlparser_duckdb.rs, tests/sqlparser_mssql.rs,
tests/sqlparser_postgres.rs: updated existing struct-literal expectations
for the new field.

Test plan

cargo test
cargo fmt --check
cargo clippy --all-targets --all-features -- -D warnings

iffyio · 2026-04-29T08:04:16Z

+    /// BigQuery `WITH CONNECTION` clause for external tables.
+    /// <https://cloud.google.com/bigquery/docs/reference/standard-sql/data-definition-language#create_external_table_statement>


Suggested change

/// BigQuery `WITH CONNECTION` clause for external tables.

/// <https://cloud.google.com/bigquery/docs/reference/standard-sql/data-definition-language#create_external_table_statement>

/// `WITH CONNECTION` clause.

/// [BigQuery](https://cloud.google.com/bigquery/docs/reference/standard-sql/data-definition-language#create_external_table_statement)

iffyio · 2026-04-29T08:04:25Z

        self.external_volume = external_volume;
        self
    }
+    /// Set the BigQuery `WITH CONNECTION` clause for external tables.


Suggested change

/// Set the BigQuery `WITH CONNECTION` clause for external tables.

/// Set the `WITH CONNECTION` clause.

iffyio · 2026-04-29T08:04:48Z

+    /// BigQuery `WITH CONNECTION` clause for external tables
+    /// <https://cloud.google.com/bigquery/docs/reference/standard-sql/data-definition-language#create_external_table_statement>


Suggested change

/// BigQuery `WITH CONNECTION` clause for external tables

/// <https://cloud.google.com/bigquery/docs/reference/standard-sql/data-definition-language#create_external_table_statement>

/// `WITH CONNECTION` clause.

/// [BigQuery](https://cloud.google.com/bigquery/docs/reference/standard-sql/data-definition-language#create_external_table_statement)

iffyio · 2026-04-29T08:05:30Z

+        // BigQuery external tables support `WITH CONNECTION <name>` and `OPTIONS(...)`
+        // clauses after the (optional) column list.


Suggested change

// BigQuery external tables support `WITH CONNECTION <name>` and `OPTIONS(...)`

// clauses after the (optional) column list.

iffyio · 2026-04-29T08:07:56Z

+        // BigQuery uses OPTIONS(...); Hive uses TBLPROPERTIES(...). They are
+        // mutually exclusive in practice, and `parse_options` returns an empty
+        // vec when the keyword isn't present, so trying OPTIONS first and
+        // falling back to TBLPROPERTIES preserves the existing Hive path
+        // without accepting both in the same statement.
+        let options = self.parse_options(Keyword::OPTIONS)?;


not sure I follow the goal here, is OPTIONS part of the WITH CONNECTION clause (if not isn't it already parsed elsewhere for bigquery today)?

Good catch, reverted. The parser change is now just the new WITH CONNECTION block plus the .with_connection(...) builder line; the fallback chain handles OPTIONS unchanged.

iffyio · 2026-04-29T08:10:14Z

+    };
+    assert_eq!(opts.len(), 1);
+
+    // Baseline: no WITH CONNECTION. The new parser branch must not produce


I think we can remove these superfluous comments from the tests

iffyio · 2026-04-29T08:11:15Z

+        "WITH CONNECTION `projects/proj/locations/us/connections/c` ",
+        r#"OPTIONS(format = "ICEBERG", uris = ["gs://b/m.json"])"#,
+    );
+    let stmts = bigquery().parse_sql_statements(sql).unwrap();


for the tests is there a reason we don't use verified_stmt? Also I think we can skip the assertions on the ast and have it solely as roundtrip tests (I don't think the ast assertions are adding much given the changes made and preexisting tests)

The pre-existing maybe_parse_options(OPTIONS) fallback already handled BigQuery OPTIONS(...) on external tables; the reorder made it unreachable without changing behavior.

feat(bigquery): parse WITH CONNECTION on CREATE EXTERNAL TABLE

4df2961

iffyio reviewed Apr 29, 2026

View reviewed changes

97nitt added 3 commits April 29, 2026 09:24

Revert OPTIONS/TBLPROPERTIES reorder in parse_create_external_table

b9b1edc

The pre-existing maybe_parse_options(OPTIONS) fallback already handled BigQuery OPTIONS(...) on external tables; the reorder made it unreachable without changing behavior.

Use verified_stmt for WITH CONNECTION tests; drop AST assertions

85c9353

Use markdown-link form for WITH CONNECTION doc comments

1ae133b

		/// BigQuery `WITH CONNECTION` clause for external tables.
		/// <https://cloud.google.com/bigquery/docs/reference/standard-sql/data-definition-language#create_external_table_statement>

	/// Set the BigQuery `WITH CONNECTION` clause for external tables.
	/// Set the `WITH CONNECTION` clause.

		// BigQuery external tables support `WITH CONNECTION <name>` and `OPTIONS(...)`
		// clauses after the (optional) column list.

Conversation

97nitt commented Apr 28, 2026

Summary

Changes

Test plan

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants